Add control_mode parameter to ControlNet #3535
Conversation
…ion and soft_injection for testing control modes
…ion and soft_injection for testing control modes
…re_prompt, more_control, even_more_control. Based on True/False combinations of internal booleans cfg_injection and soft_injection
…voke-ai/InvokeAI into feat/controlnet-control-modes
GreggHelt2
requested review from
damian0815,
lstein,
blessedcoolant,
JPPhoto,
StAlKeR7779 and
Kyle0654
as code owners
blessedcoolant
requested review from
psychedelicious and
maryhipp
as code owners
There is .. but your Cnet must have been added prior to this PR so it doesnt pick it. Delete it and readd it. The default will show up. As for the placement, I tried to put it in line with the two others .. but that just meant increasing the length of the image preview..which looked really weird. So opted for this. Also this should be a primary option. But I dont mind if it goes into advanced. Either works. Your call on that. |
|
Yep I needed to create a new cnet. The mode makes the mini view a lot less mini, and it's a big ui element for a single setting. Reluctant to make the mini view any taller, every extra item in param panel makes it less usable :/ just not an easy thing to design. I'd prefer to have it under the expanded view bc otherwise the mini view is a bit too big and cluttered. |
|
@psychedelicious It's a primary setting though. If you wanna push it down, feel free. I don't mind. But the bottom part was all processor stuff. And this one isn't. So it needs a new category along with any other future options that might be added. |
|
I attempted to resolve conflicts but rebasing isn't doing what I expected. I'll defer to @GreggHelt2 to resolve the python side, maybe it should wait until the model manager stuff is in... |
…lnet-control-modes
Only "real" conflicts were in:
invokeai/frontend/web/src/features/controlNet/components/ControlNet.tsx
invokeai/frontend/web/src/features/controlNet/store/controlNetSlice.ts
|
Okay I've got current main merged into this branch, and control_mode is working in both node UI and linear UI. |
This PR adds the "control_mode" option to ControlNet implementation.
Possible control_mode options are:
balanced, more_prompt, and more_control should be nearly identical to the equivalent options in the auto1111 sd-webui-controlnet extension
The changes to enable balanced, more_prompt, and more_control are managed deeper in the code by two booleans, "soft_injection" and "cfg_injection". The three control mode options in sd-webui-controlnet map to these booleans like:
!soft_injection && !cfg_injection ⇒ BALANCED
soft_injection && cfg_injection ⇒ MORE_CONTROL
soft_injection && !cfg_injection ⇒ MORE_PROMPT
The "unbalanced" option simply exposes the fourth possible combination of these two booleans:
!soft_injection && cfg_injection ⇒ UNBALANCED
With "unbalanced" mode it is very easy to overdrive the controlnet inputs. It's recommended to use a cfg_scale between 2 and 4 to mitigate this, along with lowering controlnet weight and possibly lowering "end step percent". With those caveats, "unbalanced" can yield interesting results.
Example of all four modes using Canny edge detection ControlNet with prompt "old man", identical params except for control_mode:
Top middle: BALANCED
Top right: MORE_CONTROL
Bottom middle: MORE_PROMPT
Bottom right : UNBALANCED
I kind of chose this seed because it shows pretty rough results with BALANCED (the default), but in my opinion better results with both MORE_CONTROL and MORE_PROMPT. And you can definitely see how MORE_PROMPT pays more attention to the prompt, and MORE_CONTROL pays more attention to the control image. And shows that UNBALANCED with default cfg_scale etc is unusable.
But here are four examples from same series (same seed etc), all have control_mode = UNBALANCED but now cfg_scale is set to 3.
And param differences are:
Top middle: prompt="old man", control_weight=0.3, end_step_percent=0.5
Top right: prompt="old man", control_weight=0.4, end_step_percent=1.0
Bottom middle: prompt=None, control_weight=0.3, end_step_percent=0.5
Bottom right: prompt=None, control_weight=0.4, end_step_percent=1.0
So with the right settings UNBALANCED seems useful.